Photography system and method including a system for lighting the scene to be shot

ABSTRACT

Provided is a method/system for taking, by an imaging device, one or more images of an object or a part of an object presented at a distance in front of the imaging device, the object being illuminated by a lighting system. The method/system may include: estimating a 3D model of the object as it will be at a predetermined time when taking the image by the imaging device; simulating, using the estimated 3D model, the illumination of the object as it will be at said time to obtain estimated images of the object in respectively a plurality of envisaged lighting modes of the lighting system; selecting a lighting mode of the lighting system from the estimated images; and controlling the lighting system to be in the selected lighting mode; and controlling the imaging device, at the time of the taking of the image of the object, to effect the taking.

The present invention relates to a photography system including a systemfor lighting the scene to be shot. It also relates to a photographymethod that can be implemented by said photography system as well as acomputer program for implementing said method.

Photography systems including a system for lighting the scene to be shotare known and mention can be made, by way of example, of biometricphotography systems in which the image of a face, of an iris or of afingerprint of a person is used for recognition purposes, and systemsfor photographing documents, such as identity documents, residenceevidence, games tickets, etc., in which the image of the document orpart thereof is used for the purpose of recognising the identity of thebearer of said document, etc. In photography systems where the inventionproves to be very effective, the position of the object to be shot isnot predetermined, as is the case with a flat scanner, where all thedocuments to be scanned are positioned against a window. This positionis therefore changing from one shot to another.

When the photography system in question includes a lighting system,because the position of the object is not predetermined, it may resultthat, in some positions of the object, reflections, in particularspecular reflections on certain areas of the object, may disturb theimaging device by creating artefacts in the image taken at the locationof these reflections. Likewise, the areas other than those that are themost strongly illuminated are in the penumbra, where the resulting lackof contrast is detrimental to a use of the image taken by the imagingdevice.

The object of the present invention is to solve this problem byincorporating, in said photography system, a lighting system that iscontrolled by a suitable control unit.

More precisely, the present invention relates to a method for taking, bymeans of an imaging device, one or more images of an object or of a partof an object presented at a distance in front of said imaging device,said object being illuminated by means of a lighting system. This methodis characterised in that it comprises the following steps:

-   -   a step of estimating a 3D model of the object as it is at a        predetermined time of the taking of the image by said imaging        device,    -   a step of simulating, using said 3D model thus estimated, the        illumination of the object as it is at said time in order to        obtain estimated images of said object in respectively a        plurality of envisaged lighting modes of said lighting system,    -   a step of selecting a lighting mode of the lighting system from        the estimated images, and    -   a step of controlling the lighting system so that it is in the        lighting mode that was selected, and    -   a step of controlling the imaging device, at said time of the        taking of the image of said object, so that said taking is        effected.

Advantageously, said step of estimating a 3D model uses a method thatuses the images taken by a 3D photography system.

According to particular embodiments of the present invention, said 3Dphotography system is a structured-light system, or consists of atime-of-flight camera.

According to another embodiment, said step of determining said 3D modeluses a stereoscopic method using pairs of views taken by said 3Dphotography system, said 3D photography system consisting of either twocameras respectively taking the views of each pair simultaneously, orone camera taking the views of each pair at two different times.

According to an advantageous embodiment, one of the two cameras or saidcamera of said 3D photography system constitutes the imaging device.

Advantageously, said step of estimating a 3D model comprises:

-   -   a step of determining a plurality of 3D models of said object on        respectively a plurality of successive occasions, and    -   a step of estimating said 3D model of the object by        extrapolation from said 3D models of said object.

The present invention also relates to a system for photographing anobject, of a type comprising an imaging device and a lighting systemprovided for illuminating said object.

According to the present invention, said photography system ischaracterised in that it also comprises a processor unit that comprises:

-   -   means for estimating a 3D model of the object as it will be at a        predetermined time T0 of the taking of the image by said imaging        device,    -   means for simulating, using said model thus estimated, the        illumination of the object as it is at said time T0 in order to        obtain estimated images of said object in respectively a        plurality of envisaged lighting modes of said lighting system,    -   means for selecting a particular lighting mode of the lighting        system from said estimated images, and    -   means for controlling said lighting system in the lighting mode        that was selected, and    -   means for controlling the imaging device, at said time T0, so        that the image of said object is taken.

According to particular embodiments, said 3D photography system is astructured-light system, or consists of a time-of-flight camera, orconsists of two cameras, said means for determining said 3D model thenusing a stereoscopic method.

Advantageously, said means for estimating said 3D model comprise:

-   -   means for determining a plurality of 3D models of said object on        respectively a plurality of successive occasions, and    -   means, using said 3D models of said object, for estimating said        3D model of the object.

Finally, the present invention relates to a program recorded on a mediumand intended to be loaded into a processing unit of a photography systemas previously described, said program comprising instructions or codeparts for implementing the steps of a method for photographing an objectin accordance with the method previously described, when said program isexecuted by said processing unit.

The features of the invention mentioned above, as well as others, willemerge more clearly from a reading of the following description of anexample embodiment, said description being given in relation to theaccompanying drawings, among which:

FIG. 1 is a block diagram of a photography system according to a firstembodiment of the invention,

FIG. 2 is a diagram illustrating the lighting simulation that is used bythe invention,

FIG. 3 is a block diagram of a photography system according to a secondembodiment of the invention,

FIG. 4 is a diagram showing the various steps implemented by a methodfor taking an image of an object according to the present invention, and

FIG. 5 is a block diagram of a computer system able to implement amethod for taking an image according to the present invention.

The photography system depicted in FIG. 1 consists of an imaging device10 that is designed to deliver image signals SI, a lighting system 20depicted schematically here in the form of two bulbs 21 and 22 forming alight source, and a control unit 30 provided for controlling the imagingdevice 10 and the lighting system 20.

According to the invention, the control unit 30 comprises an estimationunit 31 for estimating a 3D model of an object 40 as it isat the time T0of the shooting carried out by the imaging device 10.

In the embodiment depicted, the estimation unit 31 comprises a modellingunit 32 for determining, at various times t0 to tn, a 3D model of theobject 40, and an extrapolation unit 33 for determining, byextrapolation from the 3D models of the object 40 determined by themodelling unit 32, an estimated 3D model of the object 40 at time T0 ofthe shooting by the imaging device 10.

The modelling unit 32 comprises firstly a 3D photography system 34consisting, in the embodiment depicted by way of example, of two cameras341 and 342, and secondly a processing unit 35 provided for carrying outthe 3D modelling proper.

In front of the cameras 341 and 342, the object 40 is here in the formof a slightly warped flat document. This object 40 is presented at adistance in front of the imaging device 10 (that is to say presented bypassing it in front of the imaging device 10 without this object 40resting on a reference surface, such as a scanner window, and thuswithout its position being known, and in front of the cameras 341 and342. A particular point P on this document 40 is also depicted.

The processing unit 35 is designed to receive the image signals from thecameras 341 and 342 and, when an object 40 is detected in front of them(a detection made either by the control unit 30 or by any suitabledetection means), determines, from these image signals, a 3D model ofthe document 40 in front of them. The cameras 341 and 342 arecalibrated, which means that their intrinsic parameters are known andused by the processing unit 35. These intrinsic parameters are forexample given by the coefficients of a matrix K. Likewise, the extrinsicparameters of one of the cameras, for example the camera 342, withrespect to the other one in the pair, are determined and used by theprocessing unit 35.

The image I1 delivered by the camera 341 and the image I2 delivered bythe camera 342 are shown in the box of the processing unit 35. Theimages 41 and 42 of the object 40 and the images P1 and P2 of any pointon the object 40 can be seen therein respectively.

The images thus respectively formed of a point P of coordinates (x, y,z) are respectively points P1 and P2 of coordinates (u1, v1) and (u2,v2) in the respective images I1 and I2 that satisfy the followingequations:

$\begin{matrix}{{{\lambda_{1}\left\lfloor \begin{matrix}P_{1} \\1\end{matrix} \right\rfloor} = {{K\begin{bmatrix}I_{3} & 0\end{bmatrix}}\left\lfloor \begin{matrix}P \\1\end{matrix} \right\rfloor}}{and}} & (1) \\{{\lambda_{2}\left\lfloor \begin{matrix}P_{2} \\1\end{matrix} \right\rfloor} = {{K^{\prime}\begin{bmatrix}R_{12} & T_{12}\end{bmatrix}}\left\lfloor \begin{matrix}P \\1\end{matrix} \right\rfloor}} & (2)\end{matrix}$

where the matrix {R₁₂ T₁₂} (R₁₂ is a rotation matrix and T₁₂ is atranslation matrix) expresses the extrinsic parameters of the camera 342with respect to the camera 341 and λ₁ and λ₂ are unknown factorsrepresenting the fact that an infinity of antecedent points correspondto the same image point P1, P2. I3 is the unity matrix of dimensions3×3.

The images taken by the cameras 341 and 342 being given, the processingunit 35 is designed to match the image points P1 and P2 as being imagesof the same antecedent point P. This matching is known to personsskilled in the art and can be carried out by the method disclosed in thearticle by Lowe, David G. entitled “Distinctive Image Features FromScale-Invariant Keypoints” published in International Journal ofComputer Vision 60.2 (2004) p 91-110. The document by Herbert Bay, TinneTuytelaars and Luc Van Gool entitled “SURF: Speeded Up Robust Features”and published in 9th European Conference on Computer Vision, Graz,Austria, 7-13 May 2006 also mentions such a method.

The above equations show that, at each pair of image points (P1, P2)thus matched, there is a linear system of 6 equations with only 5unknowns, which are respectively the two factors λ₁ and λ₂ the threecoordinates x, y, z of the same antecedent point P of these image pointsP1 and P2. It is therefore possible, from the images supplied by thecalibrated cameras 341 and 342, to determine the coordinates x, y, z ofany antecedent point P of a pair of matched points P1, P2 and thus, byconsidering a plurality of pairs of matched points (P1, P2), todetermine a 3D model of the object 40 in front of them.

What is here referred to as a 3D model of an object is a discrete set ofpoints P of coordinates (x, y, z) that belong to the real object. Otherpoints that also belong to the real object but which do not belong tothis discrete set of points can be obtained by extrapolation of pointsof the 3D model.

As a supplement to what has been described here, it is possible toconsult the work by Richard Hartley and Andrew Zisserman entitled“Multiple View Geometry In Computer Vision”, Cambridge, 2000, inparticular for the disclosure of the above mathematical model and of thestereoscopy modelling method by means of two calibrated cameras.

The processing unit 35 is designed to control several shots of thecameras 341 and 342 at successive times t0 to tn and, for each of them,to establish, in accordance with the method disclosed above, a 3D modelof the object 40 at the corresponding time. These 3D models are suppliedto the extrapolation unit 33. This is illustrated in FIG. 1 by the narrows starting from the processing unit 35 in the direction of theextrapolation unit 33.

The extrapolation unit 33 is able to establish the movement of thisobject (translation and rotation) and to estimate the position of theobject 40 at a predetermined future time T0, which will be the time ofshooting by the imaging device 10. This position will hereinafter bereferred to as the “position at shooting time T0”. What is referred toas the “position of the object 40” is the coordinates of each of thepoints that constitute the 3D model.

The control unit 30 also comprises a simulation unit 36 that is designedto simulate the lighting of the object 40 by the lighting system 20 atthe shooting time T0, in accordance with various methods of lightingthereof. To do this, it estimates the quantity of light that each pointof the image then taken by the imaging device 10 will receive from thelighting system 20 at the shooting time T0. Various algorithms can beused to do this, such as the radiosity algorithm, analytical photometryalgorithms, the ray-tracing algorithm, etc.

In order thus to simulate the lighting of the object 40, the latter isconsidered to be situated in its position estimated by the modellingunit 31, at the time T0 of shooting by the imaging device 10.

To do this also, the lighting system 20 is calibrated, which means thatits intrinsic parameters representing the distribution of the intensityof the light in the various directions in space and the characteristicsof this light (spectrum, polarisation, etc.) as well as its extrinsicparameters (the position of the lighting system with respect to theimaging device 10 and to the 3D photographic system 34) are known andused by the simulation unit 36.

To illustrate an example of a lighting simulation method that could beused by the simulation unit 36, FIG. 2 depicts an object 40, an imagingdevice 10 in the form of a so-called pinhole model with its lens(modelled in the form of a hole or pinhole) 11 and its sensor 12 onwhich the image of the scene is formed, as well as a light source S. Thesource S emits rays in all directions in a given spatial distribution(the light intensity I varies according to the direction w of the ray inquestion: I(ω1) for the ray R1 and I(ω2) for the ray R2. For each rayemitted by it, a point of impact on the object 40 is calculated (such asthe points P and Q) in accordance with rules of geometry (intersectionof the estimated 3D model of the object 40 with the straight linecarrying the ray in question).

It is possible to limit the calculation solely to the rays that give apoint of impact on the object 40 or even solely to the rays that give apoint of impact in one or more areas of interest of the object 40. It isalso possible to limit this calculation solely to the ray, such as theray R1, that is in the emission axis of the source S, in order to limitthe computing power necessary for this processing.

According to the laws of reflection, at each point of impact thuscalculated, a plurality of rays are re-emitted, depending on the nature,diffusing or reflecting, or both at the same time, of the surface of theobject 40 at this point. For example, FIG. 2 depicts, at the points Pand Q, three rays respectively issuing from the rays R1 and R2. Thus theor each ray reflected from the point of impact with the object 40 has adirection and a light intensity that are then determined. This reflectedray, if it is captured by the imaging device 10, forms a point of impactwith the sensor 12 and provides said light intensity thereto.

Thus, in FIG. 2, the i^(th) reflected ray issuing from the ray R1,denoted R_(R1) ^(i), has a light intensity denoted I_(R1) ^(i) andimpacts the sensor 12 at the point P1 while providing thereto said lightintensity I_(R1) ^(i). Likewise, the j^(th) reflected ray issuing fromthe ray R2, denoted R_(R2) ^(i), has a light intensity denoted I_(R2)^(j) and impacts the sensor 12 at the point Q1, providing thereto saidlight intensity I_(R2) ^(j).

The total light intensity at a point of impact on the sensor 12 is afunction of the values of the light intensities for each ray reflectedon the object 40 and therefore each incident ray issuing from the sourceS.

In the simplified modelling of the pinhole camera disclosed above, onlythe rays incident on a point of the object are to be considered.

The simulation unit 36 uses this lighting simulation method for variouslighting modes of the lighting system 20. This is illustrated in FIG. 1by the m arrows that start from the simulation unit 36 towards the unit37.

According to the present description, a lighting mode is defined by aspecific combination of values attributed to the various parameters andcharacteristics of the lighting system 20, such as the spatialdistribution of the light intensity emitted in a given direction and thepoint of emission of the lighting, the spectral distribution, thepolarisation of the light emitted, etc.

For example, and as depicted in FIG. 1, a first lighting mode is givensolely by the light source 21, wherein the spatial distribution of thelight intensity is arranged inside a cone centred on a principalemission axis x21. A second lighting mode is given solely by the source22, wherein the spatial distribution of the light intensity is arrangedinside another cone centred on a principal emission axis x22 differentfrom the axis x21. A third lighting mode can be given by the two lightsources 21 and 22 simultaneously switched on.

As seen previously, the simulation unit 36 estimates, for each lightingmode and at the photographing time T0, the light intensity of points onthe sensor 12 thus forming an image, referred to as the estimated image,of the object 40 or of an area of interest thereof.

The control unit 30 also comprises a selection unit 37 for selecting,from the estimated images for various lighting modes, one of theselighting modes, on the basis of predetermined criteria,

For example, a contrario, if the lighting simulation previously carriedout, for a given lighting mode, reveals that, in this estimated image,artefacts could be created, this lighting mode is abandoned. Thelighting modes for which the light intensity at a point or a set ofpoints of the estimated image is above a threshold or below anotherthreshold, etc., could also be eliminated.

On the other hand, the lighting mode that gives the highest contrastbetween the points of lowest light intensity and those of the highestlight intensity could be chosen.

Thus the lighting mode that makes it possible both to reduce the lightintensity of the most strongly illuminated areas and to increase thelight intensity of the other areas, thus homogenising the light contrastover the whole of the estimated image, will advantageously be chosen.

The control unit 30 also comprises a control unit 38 for controlling thelighting system 20 in the lighting mode that was selected by theselection unit 37. It also comprises a control unit 39 for controlling,at the shooting time T0, the imaging device 10 for shooting the object40.

FIG. 3 depicts another embodiment of a photographing system of theinvention, according to which the imaging device 10 that does thephotographing is replaced by one of the two cameras 341 or 342 (in thiscase, the camera 342, which has an output SI for the image signals ofthe image taken at the shooting time T0 and which receives controlsignals from the control unit 39).

FIG. 4 depicts a diagram illustrating a photography method according tothe invention that is advantageously implemented by means of aphotography system, such as the one depicted in FIG. 1 or the onedepicted in FIG. 3.

Step E10 is a step of detecting an object in the scene. Step E20 is astep of estimating a 3D model of the object as it will be at apredetermined time T0 in the taking of the image by said imaging device10. Step E20 itself comprises a step E21 of determining a plurality of3D models of said object on respectively a plurality of successiveoccasions, and a step E22 of estimating said 3D model of the object byextrapolation from said 3D models of said object determined at step E21.Step E21 itself comprises a step E210 of shooting by a 3D photographysystem, such as the system 34, and a step E211 of determining the 3Dmodel of the object that was taken at step E210 by said 3D photographysystem. Steps E210 and E211 are in a loop in order to be able to berepeated a plurality of times, here n times.

Step E30 is a step of simulating the lighting of the object at the timeT0 of shooting by the imaging device 10, on the basis of the 3D modelestimated at step E22 and a given lighting mode of the lighting system20. Step E30 is in a loop in order to be able to be repeated for all theenvisaged lighting modes of the lighting system, here m in number.

Step E40 is a step of selecting a particular lighting mode of thelighting system.

Finally, step E50 is a step of controlling the lighting system in thelighting mode that was selected at step E40 and step E60 a step ofcontrolling the imaging device 10 so that the shooting is done at theshooting time T0.

This method is implemented by a control unit such as the control unit 30in FIGS. 1 and 3.

Advantageously, the units 33, 35 to 39 of this control unit 30 areimplemented by means of a computer system such as the one that is shownschematically in FIG. 5.

This computer system comprises a central unit 300 associated with amemory 301 in which there are stored firstly the code of theinstructions of a program executed by the central unit 300 and secondlydata used for executing said program. It also comprises a port 302 forinterfacing with the 3D photography system, for example with the cameras341 and 342 of FIGS. 1 and 3, a port 303 for interfacing with thelighting system, for example with the lighting system 20 in FIGS. 1 and3, and a port 304 for interfacing with the imaging device, for examplethe imaging device 10 in FIG. 1 or the camera 342 in FIG. 3. The ports302, 303 and 304 as well as the central unit 300 are connected togetherby means of a bus 305.

When the central unit 300 executes the program the instruction code ofwhich is contained in the memory 331, it implements the method that wasdescribed in relation to FIG. 4.

1. A method for taking, by means of an imaging device, one or moreimages of an object or of a part of an object presented at a distance infront of said imaging device, said object being illuminated by means ofa lighting system, wherein the method comprises the following steps: astep of estimating a 3D model of the object as it is at a predeterminedtime of the taking of the image by said imaging device, a step ofsimulating, using said 3D model thus estimated, the illumination of theobject as it is at said time in order to obtain estimated images of saidobject in respectively a plurality of envisaged lighting modes of saidlighting system, a step of selecting a lighting mode of the lightingsystem from the estimated images, and a step of controlling the lightingsystem so that it is in the lighting mode that was selected, and a stepof controlling the imaging device, at said time of the taking of theimage of said object, so that said taking is effected.
 2. The method fortaking an image according to claim 1, wherein said step of estimating a3D model uses a method that uses the images taken by a 3D photographysystem.
 3. The method for taking an image according to claim 1, whereinsaid 3D photography system is a structured-light system.
 4. The methodfor taking an image according to claim 1, wherein said 3D photographysystem consists of a time-of-flight camera.
 5. The method for taking animage according to claim 1, wherein said step of determining said 3Dmodel uses a stereoscopic method using pairs of views taken by said 3Dphotography system, said 3D photography system consisting either of twocameras taking respectively the views of each pair simultaneously, or acamera taking the views of each pair at two different times.
 6. Themethod for taking an image according to claim 3, wherein one of the twocameras or said camera of said 3D photography system constitutes theimaging device.
 7. The method for taking an image according to claim 1,wherein said step of estimating a 3D model comprises: a step ofdetermining a plurality of 3D models of said object on respectively aplurality of successive occasions, and a step of estimating said 3Dmodel of the object by extrapolation from said 3D models of said object.8. A system for taking views of an object, of a type comprising: animaging device, a lighting system designed to illuminate said object,means for estimating a 3D model of the object as it will be at apredetermined time T0 of the taking of images by said imaging device,means for simulating, using said model thus estimated, the illuminationof the object as it is at said time T0 in order to obtain estimatedimages of said object in respectively a plurality of envisaged lightingmodes of said lighting system, means for selecting a particular lightingmode of the lighting system from said estimated images, and means forcontrolling said lighting system in the lighting mode that was selected,and means for controlling the imaging device, at said time T0, so thatthe image of said object is taken.
 9. The system for taking views of anobject according to claim 8, wherein said 3D photography system is astructured-light system.
 10. The system for taking views of an objectaccording to claim 8, wherein said 3D photography system consists of atime-of-flight camera.
 11. The system for taking views according toclaim 8, wherein said 3D photography system consists of two cameras andsaid means of determining said 3D model use a stereoscopic method. 12.The system for taking views of an object according to claim 8, whereinsaid means for estimating said 3D model comprise: means for determininga plurality of 3D models of said object on respectively a plurality ofsuccessive occasions, and means, using said 3D models of said object,for estimating said 3D model of the object.
 13. A program recorded on amedium and intended to be loaded into a processing unit of a photographysystem according to claim 8, said program comprising instructions orcode parts for implementing the steps of a method for taking an image ofan object when said program is executed by said processing unit, themethod comprises the following steps: a step of estimating a 3D model ofthe object it at a predetermined time of the taking of the image by saidimaging device, a step of simulating, using said 3D model thusestimated, the illumination of the object as it is at said time in orderto obtain estimated images of said object in respectively a plurality ofenvisaged lighting modes of said lighting system, a step of selecting alighting mode of the lighting system from the estimated images, and astep of controlling the lighting system so that it is in the lightingmode that was selected, and a step of controlling the imaging device, atsaid time of the taking of the image of said object, so that said takingis effected.